List of all Crawlers
008
008 is the user-agent used by 80legs, a web crawling service provider. 80legs allows its users to design and run custom web crawls.Click on any string to get more details
008 0.83
ABACHOBot
Abacho's spider. German based portal and search engine. Has localized versions in the following countries: Austria, Switzerland, France, UK, Spain, Italy, Sweden and Turkey.Click on any string to get more details
ABACHOBot
Accoona-AI-Agent
Accoona's webcrawlerClick on any string to get more details
Accoona-AI-Agent 1.1.2
Accoona-AI-Agent 1.1.1
AddSugarSpiderBot
Click on any string to get more details
AddSugarSpiderBot
AnyApexBot
Crawler for the web directory AnyApexClick on any string to get more details
AnyApexBot 1.0
Arachmo
Japanese Crawler. Seems to be a download tool. Here's some information in japanese. If you can translate than, please let me knowClick on any string to get more details
Arachmo
B-l-i-t-z-B-O-T
Crawler for the German search engine tricus. Spiders German, Dutch, Swiss and Austrian websites. Same as BlitzBOTClick on any string to get more details
B-l-i-t-z-B-O-T
Baiduspider
Crawler for the chinese search engine BaiduClick on any string to get more details
Baiduspider 2.0
Baiduspider
- Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
- Baiduspider+(+http://www.baidu.com/search/spider.htm)
- BaiDuSpider
BecomeBot
Become crawler. Shopping related portalClick on any string to get more details
BecomeBot 3.0
BecomeBot 2.3
BeslistBot
Dutch shopping portalClick on any string to get more details
BeslistBot 1.0
BillyBobBot
Click on any string to get more details
BillyBobBot 1.0
Bimbot
Unknown crawler, gives no information. IP address belongs to Backbone Communications Inc. (BBCOM). Provides converged data and voice servicesClick on any string to get more details
Bimbot 1.0
Bingbot
Bot for Microsofts Bing search engineClick on any string to get more details
Bingbot 2.0
- Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
- Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)
BlitzBOT
Crawler for the German search engine tricus. Spiders German, Dutch, Swiss and Austrian websites. Same as B-l-i-t-z-B-O-TClick on any string to get more details
BlitzBOT
- Mozilla/4.0 (compatible; BlitzBot)
- BlitzBOT@tricus.net (Mozilla compatible)
- BlitzBOT@tricus.com (Mozilla compatible)
boitho.com-dc
Boitho's Web Crawler, a distributed crawler that downloads web pages to build the database used by Boitho.com to search in. To allow volunteers to donate their superfluous bandwidth and idle CPU time, they have developed a distributed crawler, like seti@home and Grub. That way people can install a program on their computers and help them with the crawling.Click on any string to get more details
boitho.com-dc 0.85
boitho.com-dc 0.83
boitho.com-dc 0.82
boitho.com-dc 0.81
boitho.com-dc 0.79
boitho.com-robot
This is an old version of Boitho's boitho.com-dc. It was a more traditional webrobot, run on computers controlled by Boitho, while boitho.com-dc is a distributed crawler run on the computers of volunteers.The boitho.com-robot isn’t in use any more.
Click on any string to get more details
boitho.com-robot 1.1
boitho.com-robot 1.0
btbot
btbot's search engine for bittorrents, ringtones for cell phones, friends and extraterrestrial intelligenceClick on any string to get more details
btbot 0.4
CatchBot
Web crawler for Catch, the online division of Reed Business Information AustraliaClick on any string to get more details
CatchBot 2.0
CatchBot 1.0
Cerberian Drtrs
Click on any string to get more details
Cerberian Drtrs 3.2
- Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-1)
- Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)
Charlotte
Charlotte is a spider created by Searchme, Inc. in Mountain View, CAClick on any string to get more details
Charlotte 1.1
Charlotte 1.0t
Charlotte 1.0b
- Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.searchme.com/support/)
- Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.betaspider.com/)
Charlotte 0.9t
- Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.11) Gecko/20080109 (Charlotte/0.9t; http://www.searchme.com/support/) (Charlotte/0.9t; http://www.searchme.com/support/)
- Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.11) Gecko/20080109 (Charlotte/0.9t; http://www.searchme.com/support/)
- Mozilla/5.0 (compatible; Charlotte/0.9t; http://www.searchme.com/support/)
- Mozilla/5.0 (compatible; Charlotte/0.9t; +http://www.searchme.com/support/)
ConveraCrawler
ConveraCrawler is an experimental web crawler under development since April 2004. ConveraCrawler is owned and operated by Convera CorporationClick on any string to get more details
ConveraCrawler 0.9e
ConveraCrawler 0.9d
- ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)
- ConveraCrawler/0.9d ( http://www.authoritativeweb.com/crawl)
ConveraCrawler 0.9
cosmos
Crawler from xyleme which indexes XML content on the web.Click on any string to get more details
cosmos 0.9
Covario IDS
Proprietary crawler used as part of Covario's Organic Search Insight solutionClick on any string to get more details
Covario IDS 1.0
DataparkSearch
Open source web-based search engine released under the GNU General Public License and designed to organize search within a website, group of websites, intranet or local system. DataparkSearch consists of two parts. The first part is indexing mechanism (indexer). Indexer walks over html hypertext references and stores found words and new references into database. The second part is web CGI front-end to provide search using data collected by indexer.Click on any string to get more details
DataparkSearch 4.37
DataparkSearch 4.36
DataparkSearch 4.35
- DataparkSearch/4.35-02122005 ( http://www.dataparksearch.org/)
- DataparkSearch/4.35 ( http://www.dataparksearch.org/)
DiamondBot
Crawler for Claria (formerly Gator). Adware companyClick on any string to get more details
DiamondBot
Discobot
Discobot is the experimental web crawler for Discovery EngineClick on any string to get more details
Discobot 1.0
Dotbot
Click on any string to get more details
Dotbot 1.1
Dotbot 1.0.1
EmeraldShield.com WebBot
Crawls domains as part of a spam and web filtration services. If a site is determined to contain questionable, or objectionable content it will be added to a blocklist. Ignores the robots.txt fileClick on any string to get more details
EmeraldShield.com WebBot
envolk[ITS]spider
envolk search engine spider [ITS] Internet Tracking Spider(TM)Click on any string to get more details
envolk[ITS]spider 1.6
- envolk[ITS]spider/1.6 (+http://www.envolk.com/envolkspider.html)
- envolk[ITS]spider/1.6 ( http://www.envolk.com/envolkspider.html)
EsperanzaBot
Web Crawler of Esperanza Consulting LTDClick on any string to get more details
EsperanzaBot
Exabot
Exava shopping search engine, belongs now to BecomeClick on any string to get more details
Exabot 2.0
FAST Enterprise Crawler
Product of the norvegian company Fast. Part of their FAST ProPublish solution for gathering, processing and delivering reference material to online and offline users.Click on any string to get more details
FAST Enterprise Crawler 6
- FAST Enterprise Crawler 6 used by Schibsted (webcrawl@schibstedsok.no)
- FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com/srsapp/contactus/
- FAST Enteprise Crawler/6 (www dot fastsearch dot com)
FAST-WebCrawler
Crawler for the Fast search engineClick on any string to get more details
FAST-WebCrawler 3.8
FAST-WebCrawler 3.7
- FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
- FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
FAST-WebCrawler 3.6
- FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
- FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
- FAST-WebCrawler/3.6
FAST-WebCrawler 3.x
FDSE robot
Search engine of Fluid Dynamics Software CorporationClick on any string to get more details
FDSE robot
FindLinks
A project of the Automated Speech Processing Group at the Institute of Computer Science at Universität Leipzig.Click on any string to get more details
FindLinks 2.0.1
FindLinks 1.1.6-beta6
FindLinks 1.1.6-beta4
FindLinks 1.1.6-beta1
FindLinks 1.1.5-beta7
FindLinks 1.1.4-beta1
FindLinks 1.1.3-beta9
FindLinks 1.1.3-beta8
FindLinks 1.1.3-beta6
FindLinks 1.1.3-beta4
FindLinks 1.1.3-beta2
FindLinks 1.1.3-beta1
FindLinks 1.1.2-a5
FindLinks 1.1.1-a5
FindLinks 1.1.1-a1
FindLinks 1.1.1
FindLinks 1.1-a9
FindLinks 1.1-a8
- findlinks/1.1-a8 (+http://wortschatz.uni-leipzig.de/findlinks/)
- findlinks/1.1-a8 ( http://wortschatz.uni-leipzig.de/findlinks/)
FindLinks 1.1-a7
FindLinks 1.1-a5
FindLinks 1.1-a4
FindLinks 1.1-a3
FindLinks 1.1
FindLinks 1.06
FindLinks 1.0.9
FindLinks 1.0.8
FindLinks 1.0
FurlBot
Furl's crawler. Furl is a social bookmark service from LookSmartClick on any string to get more details
FurlBot Furl Search 2.0
FyberSpider
FyberSearch web crawlerClick on any string to get more details
FyberSpider
g2crawler
g2crawler : Gnutella2Crawler codename Aenea. Not in use anymore.Click on any string to get more details
g2crawler
Gaisbot
Gais - Global Area Information Servers - Search enginge crawler of the National Chung Cheng University TaiwanClick on any string to get more details
Gaisbot 3.0+
Gaisbot 3.0
- Gaisbot/3.0+(robot05@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
- Gaisbot/3.0 (jerry_wu@openfind.com.tw; http://gais.cs.ccu.edu.tw/robot.php)
GalaxyBot
Browser for Galaxy Classifieds, a searchable directory.Click on any string to get more details
GalaxyBot 1.0
genieBot
Web-indexing robot of GenieKnows Local Search EngineClick on any string to get more details
genieBot
Gigabot
Gigablast's indexing agentClick on any string to get more details
Gigabot 3.0
Gigabot 2.0
Gigabot 1.0
Girafabot
Click on any string to get more details
Girafabot
- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322; Girafabot [girafa.com])
- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
- Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
Googlebot
Click on any string to get more details
Googlebot 2.1
- Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- Googlebot/2.1 (+http://www.googlebot.com/bot.html)
- Googlebot/2.1 (+http://www.google.com/bot.html)
Googlebot-Image
Google's image crawlerClick on any string to get more details
Googlebot-Image 1.0
GurujiBot
Indian search engineClick on any string to get more details
GurujiBot 1.0
- Mozilla/5.0 GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)
- Mozilla/5.0 GurujiBot/1.0 ( http://www.guruji.com/en/WebmasterFAQ.html)
- Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)
- GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html)
- GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)
HappyFunBot
Crawler for Happy Fun SearchClick on any string to get more details
HappyFunBot 1.1
hl_ftien_spider
Web Crawler from China. IP addresses belong to Qipusi Technology Ltd and Rongzhengwuye-ltd from Tjanjin cityClick on any string to get more details
hl_ftien_spider 1.1
hl_ftien_spider
Holmes
Sherlock Holmes is a open source universal search engine. The URL can be added by the user. Often used to spam your logfilesClick on any string to get more details
Holmes 3.9
Holmes 3.12.4
Holmes 3.12.3
Holmes 3.12.2
Holmes 3.12.1
htdig
Crawler of the ht://Dig Group's software package, a system for indexing and searching a finite (not necessarily small) set of sites or intranet. It is not meant to replace any of the many internet-wide search engines. htdig retrieves HTML documents using the HTTP protocol.Click on any string to get more details
htdig 3.1.6
htdig 3.1.5
- htdig/3.1.5 (webmaster@online-medien.de)
- htdig/3.1.5 (root@localhost)
- htdig/3.1.5 (infosys@storm.rmi.org)
- htdig/3.1.5
iaskspider
Bot for iAsk , chinese search engine from Sina.comClick on any string to get more details
iaskspider 2.0
iaskspider
ia_archiver
Alexa Web crawlerClick on any string to get more details
ia_archiver 8.9
- ia_archiver/8.9 (Windows NT 3.1; en-US;)
- ia_archiver/8.9 (Windows 3.9; en-US;)
- ia_archiver/8.9 (Linux 1.0; en-US;)
ia_archiver 8.8
ia_archiver 8.2
ia_archiver 8.1
ia_archiver 8.0
ia_archiver
iCCrawler
ICCrawler is ICCenter's specialized web-crawling robot. Currently they are collecting only job offers from company sites. Those job offers are getting listed at ICjobsClick on any string to get more details
iCCrawler
ichiro
Japanese Webcrawler for GooClick on any string to get more details
ichiro 4.0
ichiro 3.0
ichiro 2.0
- ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html)
- ichiro/2.0 (ichiro@nttr.co.jp)
- ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
igdeSpyder
Crawler for the russian IGDE commercial search engineClick on any string to get more details
igdeSpyder
IRLbot
IRL-crawler is a Texas A&M University research project sponsored in part by the National Science Foundation that investigates algorithms for mapping the topology of the Internet and discovering the various parts of the web. The crawler downloads random web pages (text only) and follows certain links to find other websites.Click on any string to get more details
IRLbot 3.0
- IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler/)
- IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
IRLbot 2.0
- IRLbot/2.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
- IRLbot/2.0 (+http://irl.cs.tamu.edu/crawler)
- IRLbot/2.0 ( http://irl.cs.tamu.edu/crawler)
IssueCrawler
Govcom.org Foundation's web bot. Locates and visualizes networks on the Web. The Issue Crawler is used by NGOs and other researchers to answer questions about specific networks and effective networking more generally. You also may do in-depth research with the software. You need an account to use it.Click on any string to get more details
IssueCrawler
Jaxified Bot
Click on any string to get more details
Jaxified Bot
Jyxobot
Czech Webcrawler for JyxoClick on any string to get more details
Jyxobot 1
KoepaBot
Click on any string to get more details
KoepaBot
L.webis
Crawler developed at the Institute of Informatics and Telematics (IIT), of the National Research Council (CNR) of Italy, in PisaClick on any string to get more details
L.webis 0.87
LapozzBot
Hungarian bot. Spiders for the Lapozz search engine.Üdvözlöm !?!
Click on any string to get more details
LapozzBot 1.4
Larbin
Multi-purpose web crawlerClick on any string to get more details
Larbin 5.0
Larbin 2.6.3
- larbin_2.6.3 zumesun@hotmail.com
- larbin_2.6.3 tangyi858@163.com
- larbin_2.6.3 ltaa_web_crawler@groupes.epfl.ch
- larbin_2.6.3 larbin2.6.3@unspecified.mail
- larbin_2.6.3 gqnmgsp@ruc.edu.cn
- larbin_2.6.3 ghary@sohu.com
- larbin_2.6.3 capveg@cs.umd.edu
- larbin_2.6.3 (wgao@genieknows.com)
- larbin_2.6.3 (ltaa_web_crawler@groupes.epfl.ch)
- larbin_2.6.3 (larbin@behner.org)
- larbin_2.6.3 (larbin2.6.3@unspecified.mail)
Larbin 2.6.2
- larbin_2.6.2 vitalbox1@hotmail.com
- larbin_2.6.2 pierre@micro-fun.ch
- larbin_2.6.2 listonATccDOTgatechDOTedu
- larbin_2.6.2 larbin@correa.org
- larbin_2.6.2 larbin2.6.2@unspecified.mail
- larbin_2.6.2 kalou@kalou.net
- larbin_2.6.2 dthunen@princeton.edu
- larbin_2.6.2 (vitalbox1@hotmail.com)
- larbin_2.6.2 (pierre@micro-fun.ch)
- larbin_2.6.2 (larbin@correa.org)
- larbin_2.6.2 (larbin2.6.2@unspecified.mail)
Larbin 2.6.1
Larbin 2.5.0
Larbin xy250
Larbin
LDSpider
LDSpider project aims to build a web crawling framework for the linked data webClick on any string to get more details
LDSpider
LexxeBot
Bot for Lexxe Search EngineClick on any string to get more details
LexxeBot 1.0
Linguee Bot
Search engine for bilingual texts. Helps with translating common phrases into another languageClick on any string to get more details
Linguee Bot
LinkWalker
SEVENtwentyfour Inc Link CheckerClick on any string to get more details
LinkWalker 2.0
LinkWalker
lmspider
Collects text from the web as part of a research project at Scansoft (renamed Nuance) ,trying to use web documents to improve the linguistic models used in their speech recognition engineClick on any string to get more details
lmspider
lwp-trivial
lwp-trivial is the user-agent associated with the Perl code Module LWP::SimpleClick on any string to get more details
lwp-trivial 1.41
lwp-trivial 1.38
lwp-trivial 1.36
lwp-trivial 1.35
lwp-trivial 1.33
mabontland
Crawler for the web directory mabontlandClick on any string to get more details
mabontland
magpie-crawler
Crawler for BrandwatchClick on any string to get more details
magpie-crawler 1.1
masscan-ng
Click on any string to get more details
masscan-ng 1.3
Mediapartners-Google
Unregistered versions of opera prior to 8.5 contained advertising. To serve up relevant adverts based on what you are browsing Google provided these adverts.More information
Click on any string to get more details
Mediapartners-Google 2.1
MJ12bot
Majestic-12 Web CrawlerClick on any string to get more details
MJ12bot 1.2.4
MJ12bot 1.2.3
MJ12bot 1.0.8
MJ12bot 1.0.7
MJ12bot 1.0.6
MJ12bot 1.0.5
Mnogosearch
Web search engine software for intranet and internet servers from Mnogosearch.org (a project of Lavtech)Click on any string to get more details
Mnogosearch 3.1.21
mogimogi
Unclear. The IP address belongs to Goo but they don't give any information about that bot. Goo itself uses ichiro for their search engineClick on any string to get more details
mogimogi 1.0
MojeekBot
MojeekBot (formerly Citenikbot) is the web crawler for the Mojeek search engine.Click on any string to get more details
MojeekBot 2.0
MojeekBot 0.2
Moreoverbot
Rssfeed botClick on any string to get more details
Moreoverbot 5.1
Moreoverbot 5.00
- Moreoverbot/5.00 (+http://www.moreover.com; webmaster@moreover.com)
- Moreoverbot/5.00 (+http://www.moreover.com)
Morning Paper
Crawler for Boutell.com.Click on any string to get more details
Morning Paper 1.0
msnbot
MSN (or Microsoft Service Network) Search Web CrawlerClick on any string to get more details
msnbot 2.1
msnbot 2.0b
msnbot 1.1
msnbot 1.0
msnbot 0.9
msnbot 0.11
msnbot 0.1
MSRBot
Microsoft Research web crawlerClick on any string to get more details
MSRBot
MVAClient
I have no information about this one. The ip address belongs to Chunghwa Telecom Co.,Ltd. in Taiwan. It is blacklisted by SORBS. If you know anything about this bot please let me knowClick on any string to get more details
MVAClient
mxbot
Crawler for ChainnClick on any string to get more details
mxbot 1.0
- Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)
- Mozilla/5.0 (compatible; mxbot/1.0; http://www.chainn.com/mxbot.html)
NetResearchServer
Spider for LOOP Improvements. Crawls the web by using the links found in the DMOZ Open Directory Project.Click on any string to get more details
NetResearchServer 4.0
NetResearchServer 3.5
NetResearchServer 2.8
NetResearchServer 2.7
NetResearchServer 2.5
NetResearchServer
NetSeer Crawler
Click on any string to get more details
NetSeer Crawler 2.0
NewsGator
Click on any string to get more details
NewsGator 2.5
NewsGator 2.0
NG-Search
NG-Search is experimental searchengine with new semantic trials to list the most relevance words and groups around your queryClick on any string to get more details
NG-Search 0.9.8
NG-Search 0.86
nicebot
Click on any string to get more details
nicebot
noxtrumbot
Spanish search engine for Spanish and Portuguese pages. Belongs to TPI, Telefónica Publicidad e Información, S.AClick on any string to get more details
noxtrumbot 1.0
Nusearch Spider
Crawls for the Nusearch search engine. Customizable search engine with some additional features like active bookmarks, and alternative result views.Click on any string to get more details
Nusearch Spider
NutchCVS
Open source robotClick on any string to get more details
NutchCVS 0.8-dev
NutchCVS 0.7.2
NutchCVS 0.7.1
- NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
- NutchCVS/0.7.1 (Nutch running at UW; http://crawlers.cs.washington.edu/; sycrawl@cs.washington.edu)
NutchCVS 0.7
NutchCVS 0.06-dev
- NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)
- NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; jagdeepssandhu@hotmail.com)
NutchCVS 0.05
Nymesis
Click on any string to get more details
Nymesis 1.0
obot
German spider from Cobion, now part of Internet Security Systems. Scans the web for their clients looking for copyright infringementClick on any string to get more details
obot
oegp
The IP address belongs to the Deutsche Telekom in Germany. They don't give any information about that crawler. IP address is blacklistedClick on any string to get more details
oegp 1.3.0
omgilibot
Click on any string to get more details
omgilibot 0.4
omgilibot 0.3
OmniExplorer_Bot
New crawler for Omni-Explorer. Site not launched yet (February 06)Click on any string to get more details
OmniExplorer_Bot 6.70
OmniExplorer_Bot 6.65a
OmniExplorer_Bot 6.63b
OmniExplorer_Bot 6.62
OmniExplorer_Bot 6.60
OmniExplorer_Bot 6.47
OmniExplorer_Bot 5.91c
OmniExplorer_Bot 5.28
OmniExplorer_Bot 5.25
OmniExplorer_Bot 5.20
OmniExplorer_Bot 5.01
OmniExplorer_Bot 4.80
OmniExplorer_Bot 4.32
OOZBOT
Click on any string to get more details
OOZBOT 0.20
OOZBOT 0.17
Orbiter
Spider for DailyOrbit search engine. Visits only the homepage of a domain.Click on any string to get more details
Orbiter
PageBitesHyperBot
Crawler for PageBites, a search engine for job openings and/or résumés. You can also post your résumé and/or job opening to them.Click on any string to get more details
PageBitesHyperBot 600
Peew
Click on any string to get more details
Peew 1.0
polybot
Polybot is a distributed web crawler developed in the Department of Computer and Information Science at Polytechnic University as part of an academic research project that explores new techniques for searching and analyzing the World Wide Web. This bot is not connected to the search site www.polybot.com and it has nothing to do with the virus of the same nameClick on any string to get more details
polybot 1.0
Pompos
Crawler for the french search engine dirClick on any string to get more details
Pompos 1.3
Pompos 1.2
Pompos 1.1
PostPost
PostPost is a social search engine. They crawl pages that are linked to in Tweets, index key portions of them, and associate them to their parent Tweets.Click on any string to get more details
PostPost 1.0
Psbot
Image crawler from Picsearch. Indexes images from the webClick on any string to get more details
Psbot 0.1
PycURL
PycURL is a Python interface to libcurl. PycURL can be used to fetch objects identified by a URL from a Python program, similar to the urllib Python moduleClick on any string to get more details
PycURL 7.23.1
PycURL 7.19.7
PycURL 7.19.5
PycURL 7.19.3
PycURL 7.19.0
PycURL 7.18.2
PycURL 7.18.0
PycURL 7.16.4
PycURL 7.15.5
PycURL 7.13.2
PycURL
Qseero
Click on any string to get more details
Qseero 1.0.0
Radian6
Click on any string to get more details
Radian6
RAMPyBot
RAMPyBot is giveRamp's (give Relevant Answers with Meticulous Precision) spider. Belongs to GomventsClick on any string to get more details
RAMPyBot 0.1
RufusBot
Web Crawler from WebarooClick on any string to get more details
RufusBot
SandCrawler
This one belongs to Microsoft. No idea what they are spidering with this one.Click on any string to get more details
SandCrawler
SBIder
SiteSell web crawlerClick on any string to get more details
SBIder 0.8-dev
ScoutJet
ScoutJet is the web crawler for blekko, a Silicon Valley based search engine created by the founders of DMOZ and TopixClick on any string to get more details
ScoutJet
Scrubby
Scrub the web's crawlerClick on any string to get more details
Scrubby 2.2
- Scrubby/2.2 (http://www.scrubtheweb.com/)
- Mozilla/5.0 (compatible; Scrubby/2.2; +http://www.scrubtheweb.com/)
- Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)
Scrubby 2.1
- Scrubby/2.1 (http://www.scrubtheweb.com/)
- Mozilla/5.0 (compatible; Scrubby/2.1; +http://www.scrubtheweb.com/abs/meta-check.html)
SearchSight
Search engine and directoryClick on any string to get more details
SearchSight 2.0
Seekbot
Spider for the european seekport search engine.Click on any string to get more details
Seekbot 1.0
- Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2
- Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/2.1
- Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3
- Seekbot/1.0 (http://www.seekbot.net/bot.html)
semanticdiscovery
The Semantic Discovery robot collects content from the web to be matched into focused "product and service" taxonomies and then published in multiple search engine directories.Click on any string to get more details
semanticdiscovery 0.1
Sensis Web Crawler
Click on any string to get more details
Sensis Web Crawler
SEOChat::Bot
Click on any string to get more details
SEOChat::Bot 1.1
SeznamBot
Click on any string to get more details
SeznamBot 2.0
Shim-Crawler
Japanese crawler, collects web pages for researches related to web-search and data mining. The Crawler is used by the members of Chikayama-Taura Laboratory to crawl web-pages only for the research purposes.Click on any string to get more details
Shim-Crawler
- Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; crawl@logos.ic.i.u-tokyo.ac.jp)
- Shim-Crawler
ShopWiki
ShopWiki is a shopping search engine crossed with a wiki. The service crawls the web for product listings, then allows users to write reviews in a collaborative wiki format.Click on any string to get more details
ShopWiki 1.0
Shoula robot
Shoula Search EngineClick on any string to get more details
Shoula robot
silk
Web crawler for the Slider DMOZ search engine. Crawls DMOZ entries only. You can add your own site by including a Slider.com search box or button on the main page of your website or by paying.Click on any string to get more details
silk 1.0
Sitebot
Click on any string to get more details
Sitebot 0.1
- Mozilla/5.0 (compatible; SiteBot/0.1; +http://www.sitebot.org/robot/)
- Mozilla/5.0 (compatible; SiteBot/0.1; http://www.sitebot.org/robot/)
Snappy
UrlTrends' robot for generating ReportsClick on any string to get more details
Snappy 1.1
sogou spider
Chinese spider. Sohu's proprietary search engine, Sogou, which means ‘Search Dog’ in Chinese, initially launched in August 2004Click on any string to get more details
sogou spider
Sosospider
Crawler for Soso (搜搜), a Chinese search engine owned by Tencent Holdings LimitedClick on any string to get more details
Sosospider
Speedy Spider
Speedy is an automated web crawler used to build the search engine index at EntirewebClick on any string to get more details
Speedy Spider 5.0
- Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
- Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)
Speedy Spider 1.3
Speedy Spider 1.2
Speedy Spider 1.1
Speedy Spider 1.0
- Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/speedyspider/)
- Speedy Spider (Beta/1.0; www.entireweb.com)
Speedy Spider
- Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
- Speedy Spider (http://www.entireweb.com/about/search_tech/speedyspider/)
- Speedy Spider (http://www.entireweb.com)
Sqworm
Click on any string to get more details
Sqworm 2.9.85-BETA
StackRambler
Russian spider. Crawls the web for Ramber.ru, a search engine of Rambler MediaClick on any string to get more details
StackRambler 2.0
suggybot
German crawlerClick on any string to get more details
suggybot 0.01a
SurveyBot
Monitors internet statistics for the Whois Source domain search engineClick on any string to get more details
SurveyBot 2.3
SynooBot
Spider for the German web directory Synoo. Accepts free submissions of German websitesClick on any string to get more details
SynooBot 0.7.1
Teoma
The Teoma Crawler is Ask Jeeves' Web-indexing robotClick on any string to get more details
Teoma
- Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)
- Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
- Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
TerrawizBot
Indian search engine bot. TerrawizBot is the user-agent for Terrawiz's web crawler. Terrawiz is a privately held startup with development center in Bangalore, India.Click on any string to get more details
TerrawizBot 1.0
TheSuBot
Hyro-Mediaservice German crawlerClick on any string to get more details
TheSuBot 0.2
TheSuBot 0.1
Thumbnail.CZ robot
Takes screenshots of websites. Thumbnail.CZ visits the page and checkes if it exists. Later a Konqueror browser visits the site and takes a screenshot of the Homepage. Doesn't follow any links and doesn't obey robots.txt. Provides search engines and catalogues with thumbnail previews of websites to make results more attractive.Click on any string to get more details
Thumbnail.CZ robot 1.1
TinEye
The TinEye crawler is a web crawler for an open image search project currently being builtClick on any string to get more details
TinEye 1.1
TinEye
truwoGPS
Click on any string to get more details
truwoGPS 1.0
TurnitinBot
Turnitin.com's web crawling robot. This robot collects content from the Internet for the sole purpose of helping educational institutions prevent plagiarism. In particular, they compare student papers against the content they find on the Internet to see if they can find similarities.Click on any string to get more details
TurnitinBot 2.1
TurnitinBot 2.0
TurnitinBot 1.5
- TurnitinBot/1.5 http://www.turnitin.com/robot/crawlerinfo.html
- TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)
- TurnitinBot/1.5 http://www.turnitin.com/robot/crawlerinfo.html
- TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)
TweetedTimes Bot
The Tweeted Times aggregates news in a Twitter stream and ranks them by popularity among ones friendsClick on any string to get more details
TweetedTimes Bot 1.0
TwengaBot
Click on any string to get more details
TwengaBot
updated
Spider for the product search engine Updated.Click on any string to get more details
updated 0.1-beta
Urlfilebot
Click on any string to get more details
Urlfilebot 2.2
Vagabondo
The WiseGuys webagent visits primarily dutch (.nl) websites, WAP sites however are visited worldwide. An alternate identification is: Bilbo . This is an old webagent, that is mainly still in use for WAP. These robots mainly gather data for parties including Track.nl and Kobala.nl.Click on any string to get more details
Vagabondo 4.0Beta
Vagabondo 2.2
Vagabondo 2.1
Vagabondo 2.0 MT
VoilaBot
French telecom's Voila search engine crawlerClick on any string to get more details
VoilaBot 1.2
Vortex
Vortex Web Indexing Robot, part of a study on internet link distributionClick on any string to get more details
Vortex 2.2
- Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/)
- Vortex/2.2 ( http://marty.anstey.ca/robots/vortex/)
Vortex 1.2
voyager
voyager is Cosmix Corporation's web crawling robot.It fetches documents from the web to build the index for the Kosmix search engineClick on any string to get more details
voyager 2.0
voyager 1.0
VYU2
Click on any string to get more details
VYU2
webcollage
WebCollage is a program that creates collages out of random images found on the Web. More images are being added to the collage about once a minute, so this page will reload itself periodically. Clicking on one of the images in the collage will take you to the page on which it was found.Click on any string to get more details
webcollage 1.93
webcollage 1.129
webcollage 1.125
webcollage 1.117
webcollage 1.114
Websquash.com
Websquash web crawlerClick on any string to get more details
Websquash.com
wf84
WebFountain™ is a set of research technologies that collect, store and analyze massive amounts of unstructured and semi-structured text. It is built on an open, extensible platform that enables the discovery of trends, patterns and relationships from data. For more information on text analytics research at IBM, please visit UIMA.Click on any string to get more details
wf84
WoFindeIch Robot
Crawler for the Switzerland based wofindeich search engine. For .ch and .li Domains only, unless they have Switzerland related content.Click on any string to get more details
WoFindeIch Robot 1.0
- WoFindeIch Robot 1.0(+http://www.search.wofindeich.com/robot.php)
- WoFindeIch Robot 1.0( http://www.search.wofindeich.com/robot.php)
WomlpeFactory
Not clear what this bot is doingClick on any string to get more details
WomlpeFactory 0.1
Xaldon_WebSpider
Xaldon Technologies's Webspider crawls the web and copies websites to your harddisk for offline browsingClick on any string to get more details
Xaldon_WebSpider 2.0.b1
yacy
yacy is a client to the YaCy P2P-based Web indexing network. It can crawl the web, search the web and provide web-services like a web server, file share, a wiki and peer-to-peer messages.Click on any string to get more details
yacy
- yacybot (x86 Windows XP 5.1; java 1.6.0_12; Europe/de) http://yacy.net/bot.html
- yacybot (x86 Windows XP 5.1; java 1.6.0_11; Europe/de) http://yacy.net/bot.html
- yacybot (x86 Windows XP 5.1; java 1.6.0; Europe/de) http://yacy.net/yacy/bot.html
- yacybot (x86 Windows 2000 5.0; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
- yacybot (ppc Mac OS X 10.5.2; java 1.5.0_13; Europe/de) http://yacy.net/bot.html
- yacybot (ppc Mac OS X 10.4.10; java 1.5.0_07; Europe/de) http://yacy.net/bot.html
- yacybot (i386 Mac OS X 10.5.7; java 1.5.0_16; Europe/de) http://yacy.net/bot.html
- yacybot (i386 Linux 2.6.9-023stab046.2-smp; java 1.6.0_05; Europe/en) http://yacy.net/bot.html
- yacybot (i386 Linux 2.6.8-022stab070.5-enterprise; java 1.4.2-03; Europe/en) yacy.net
- yacybot (i386 Linux 2.6.31-16-generic; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
- yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
- yacybot (i386 Linux 2.6.24-28-generic; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
- yacybot (i386 Linux 2.6.24-24-generic; java 1.6.0_07; Europe/en) http://yacy.net/bot.html
- yacybot (i386 Linux 2.6.24-23-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
- yacybot (i386 Linux 2.6.23; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
- yacybot (i386 Linux 2.6.23; java 1.6.0_04; Europe/en) http://yacy.net/bot.html
- yacybot (i386 Linux 2.6.22-14-generic; java 1.6.0_03; Europe/de) http://yacy.net/bot.html
- yacybot (amd64 Windows 7 6.1; java 1.6.0_17; Europe/de) http://yacy.net/bot.html
- yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
- yacybot (amd64 Linux 2.6.16-2-amd64-k8-smp; java 1.5.0_10; Europe/en) http://yacy.net/yacy/bot.html
More yacy user agents strings -->>
Yahoo! Slurp
Yahoo! Slurp - Yahoo!'s Web CrawlerClick on any string to get more details
Yahoo! Slurp
Yahoo! Slurp China
Yahoo's crawler for ChinaClick on any string to get more details
Yahoo! Slurp China
YahooSeeker
Click on any string to get more details
YahooSeeker 1.2
YahooSeeker-Testing
Click on any string to get more details
YahooSeeker-Testing 3.9
YandexBot
Russian botClick on any string to get more details
YandexBot 3.0
YandexImages
Click on any string to get more details
YandexImages 3.0
Yasaklibot
Click on any string to get more details
Yasaklibot 1.2
Yeti
Yeti is the crawler for Naver, a popular Korean search engineClick on any string to get more details
Yeti 1.0
YodaoBot
Crawler for the Chinese search engine YoudaoClick on any string to get more details
YodaoBot 1.0
yoogliFetchAgent
Yoogli search engine (under construction). Should come live february 2006Click on any string to get more details
yoogliFetchAgent 0.1
YoudaoBot
Youdoa is a chinese search engine from NetEaseClick on any string to get more details
YoudaoBot 1.0
Zao
Japanese crawler of Kototoi.orgClick on any string to get more details
Zao 0.1
Zealbot
Crawls all Web sites listed with LookSmart each week to ensure that they are still active, responsive sites.Click on any string to get more details
Zealbot 1.0
zspider
Crawler for Redkolibri, a new search engine under development, to be released in 2006.Click on any string to get more details
zspider 0.9-dev
ZyBorg
LookSmart's WiseNut search engine crawlerClick on any string to get more details
ZyBorg 1.0
- Mozilla/4.0 compatible ZyBorg/1.0 DLC (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
- Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
- Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com)
- Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
- Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnutbot.com)
- Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net; http://www.WISEnutbot.com)